Method and apparatus for correcting geometric and optical aberrations in augmented reality

ABSTRACT

The disclosed technology teaches an AR calibration system for compensating for AR headset distortions. A calibration image is provided to an external screen and viewable through a headset reflector, and an inverse of the calibration image is provided to a headset display, reflected off the reflector and observed by a camera of the system while it is simultaneously observing the calibration image on the external screen. The camera is located to represent a user&#39;s point of view and aligned to observe the inverse calibration image projected onto the reflector. A distortion mapping transform is created using an algorithm to search through projection positions of the inverse calibration image until the inverse image observed by the camera cancels out an acceptable portion of the calibration image provided to the external screen as observed through the reflector by the camera, and the transform is used by the headset, to compensate for distortions.

RELATED APPLICATION

This application claims the benefit of U.S. Provisional PatentApplication No. 62/756,053, entitled, “METHOD AND APPARATUS FORCORRECTING GEOMETRIC AND OPTICAL ABERRATIONS IN AUGMENTED REALITY”,filed Nov. 5, 2018. This U.S. Provisional patent application is herebyincorporated by reference for all purposes.

ADDITIONAL INCORPORATIONS

U.S. patent application Ser. No. 14/626,898, entitled “SYSTEMS ANDMETHODS OF CREATING A REALISTIC GRAB EXPERIENCE IN VIRTUALREALITY/AUGMENTED REALITY ENVIRONMENTS,” filed on Feb. 19, 2015, nowU.S. Pat. No. 9,696,795, issued Jul. 4, 2017 is hereby incorporated byreference for all purposes as if fully set forth herein.

FIELD OF THE TECHNOLOGY DISCLOSED

The technology disclosed relates generally to special calibrationtechniques for compensating for augmented reality (AR) headsets. Morespecifically the disclosed technology relates to calibrating AR headsetsto account for distortion caused by manufacturing deviations,differences in user head shape and eye location and shifting of the ARheadset during use.

BACKGROUND

The subject matter discussed in this section should not be assumed to beprior art merely as a result of its mention in this section. Similarly,a problem mentioned in this section or associated with the subjectmatter provided as background should not be assumed to have beenpreviously recognized in the prior art. The subject matter in thissection merely represents different approaches, which in and ofthemselves can also correspond to implementations of the claimedtechnology.

Augmented Reality (AR) technology refers to real time registration of 2Dor 3D computer generated imagery onto a live view of a real-worldphysical space. When using AR technology, a user is able to view andinteract with the augmented imagery in such a way as to manipulate thevirtual objects in their view.

Users of AR headsets and the like are come in all shapes and sizes. Forexample, the head shape and size of different users can varydramatically. Further, eye location and interpupillary distance (IPD)can very between each user. Additionally, AR headsets are manufacturedbased on an average person's head shape, head size, eye location andIPD, leaving the user with an AR headset that is designed for theaverage person. Also, during the manufacturing process there are oftenmanufacturing deviations that can result in AR headsets having differentalignments, etc. Furthermore, during use of the AR headset, the headsetcan be moved on the user's head, thus changing the alignment of the ARheadset during use. In order to provide the best possible AR experience,the AR headset should be calibrated to compensate for thesemanufacturing deviations, the physical differences of each user and theshifting of the AR headset on the user's head during use.

Calibration and alignment are more challenging in AR than in virtualreality (VR). In VR, users' brains often gloss over mismatches in timeand space, because the user has nothing to visually compare them to. InAR, one can see the virtual and real worlds simultaneously. Unlike inVR, any spatiotemporal mismatch can be visually compared against thereal world, which is an unforgiving standard that requires a high degreeof accuracy. For AR, that accuracy needs to be maintained across a muchwider field of view than for earlier-generation AR headset designs. ForAR headsets with a wide field of vision (FoV), calibration and alignmentare more challenging than for headsets with a narrow FoV, becauseheadsets with a wide FoV have optics that create a stereo-divergentoff-axis distortion that cannot be accurately modelled with conventionalradial polynomials.

3D printing is an ongoing revolution in low volume manufacturing of ARheadsets. While using 3D printing for generating AR headsets iseconomical, it comes at the expense of mechanical deviations, asmentioned above. Optics generally require extremely tight tolerances.Given the choice, optic designers prefer their designs to be machinedfrom solid granite. Given the weight and cost associated with granite,they are forced to solve the problem some other way.

Consequently, there is a need for using an AR calibration system tocompensate for distortions caused by manufacturing deviations in an ARheadset and to compensate for physical differences in users and shiftingof the AR headset during use.

BRIEF DESCRIPTION OF THE DRAWINGS

10891 In the drawings, like reference characters generally refer to likeparts throughout the different views. Also, the drawings are notnecessarily to scale, with an emphasis instead generally being placedupon illustrating the principles of the technology disclosed. In thefollowing description, various implementations of the technologydisclosed are described with reference to the following drawings. Thepatent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee. The color drawings also may be available in PAIRvia the Supplemental Content tab.

FIG. 1 illustrates an augmented reality (AR) headset and a block diagramof components of the AR headset according to an implementation of thetechnology disclosed.

FIG. 2 illustrates a block diagram of a calibration and alignmentsystem, an external display screen, and a top view of the AR headsetillustrated in FIG. 1, according to an implementation of the technologydisclosed.

FIG. 3A illustrates an example calibration image as displayed by anexternal display screen.

FIG. 3B illustrates an example inverted calibration image as projectedonto left and right reflectors of an AR headset.

FIG. 4A illustrates an example inverse calibration image as reflected byleft-eye and right-eye reflectors of an AR headset before calibration.

FIG. 4B illustrates an example inverted calibration image as reflectedby left-eye and right-eye reflectors of the AR headset aftercalibration.

FIG. 5A illustrates an example calibration image as displayed by anexternal display screen.

FIG. 5B illustrates an example inverted calibration image as projectedonto left and right reflectors of an AR headset after a calibration hasbeen performed.

FIG. 6 illustrates actual before and after calibration images as viewedby a camera through a reflector of an AR headset.

FIG. 7 and FIG. 8 list an example distortion mapping transform,expressed in JSON that gets loaded onto an AR headset.

FIG. 9 illustrates an example distortion mapping transform for aspecific AR headset including specific metadata for the specific ARheadset.

FIG. 10 illustrates an example computing system that can comprise one ormore of the elements shown in FIGS. 1 and 2.

DETAILED DESCRIPTION

The following detailed description is made with reference to thefigures. Sample implementations are described to illustrate thetechnology disclosed, not to limit its scope, which is defined by theclaims. Those of ordinary skill in the art will recognize a variety ofequivalent variations on the description that follows.

Manufacturers of augmented reality (AR) headsets need an AR calibrationsystem that can compensate for distortions caused by manufacturingdeviations in the AR headsets as well as differences in the head shapeof a user (e.g., interpupillary differences (IPDs) and movements of theAR headset with respect to the user's head during use. In static opticalsystems, it usually is not worth going through the trouble ofdetermining per-headset optical models for distortion correction.However, near-eye displays are anything but static. Also, as mentionedabove, eye positions change due to different peoples' head sizes, IPDsand headset ergonomics, as well as the gradual shift of the headset onthe head over a session. Any one of these factors alone can hamper theillusion of AR.

The disclosed AR calibration system includes calibration techniques for(i) creating a distortion mapping transform to compensate fordistortions in AR headsets caused by manufacturing deviations and (ii)updating and/or creating a new distortion mapping transform tocompensate for distortions in AR headsets caused by users havingdifferent head shapes and IPDs as well as shifting of the AR headsetduring use thereof by using eye tracking calibration techniques. Thesecalibration techniques enable improved user experience, greater safetyand greater functionality to users of AR headsets for gaming, machinecontrol and/or machine communications applications.

The AR visual illusion created by wide-field-of-view displays and opticsmotivates the need for the disclosed calibration and distortion mappingtransform for generating coherent images for the wearer's eyes. Thebenefits of the technology disclosed include solving for physicallyaccurate parameters with a system-in-the-loop approach that allows forextrapolating new viewpoints for distortion compensation. The technologydisclosed is unaffected by the calibration of the measurement device,which increases accuracy and avoids local minima that single camerasystems get caught in, so it remains valid even at new or different viewpositions than the view positions used for calibration. Next, an exampleAR calibration system is described for calibrating an AR headset.

FIG. 1 illustrates an AR headset 100 and a block diagram 114 ofcomponents of the AR headset 100 according to an implementation of thetechnology disclosed. Specifically, FIG. 1 illustrates an example ARheadset 100 that uses the disclosed AR calibration system to compensatefor distortions caused by manufacturing deviations of the AR headset100. The AR headset 100 can be a personal head mounted display (HMD)having a goggle form factor such as shown in FIG. 1. The disclosedcalibration system can compensate for distortions that originate due toheadset manufacturing deviations. Similar techniques can be implemented,as discussed below, to compensate for differences in head shape/size andIPDs of different users.

AR headset 100 includes two displays 102, 104, one for the left eye ofthe user and one for the right eye of the user. Display 102 can bereferred to as a left-eye display and display 104 can be referred to asa right-eye display. These displays 102, 104 face toward reflectors 106,108 located in front of the user when the AR headset 100 is worn by theuser. Just like the displays 102, 104, reflector 106 can be referred toas a left-eye reflector and reflector 108 can be referred to as aright-eye reflector. The reflectors 106, 108 respectively reflect thelight coming from the displays 102, 104, and into the wearer's eyes.

AR headset 100 also includes rotation adjustment 110 for changing analignment of the AR headset 100 while on the user's head, and a slideadjustment 112 for changing a distance of the reflectors 106, 108 fromthe wearer's eyes.

Additionally, the AR headset 100 includes a system 114 includinghardware and software for driving the displays 102, 104 as well astracking the eyes of the user. Specifically, the system 114 includes anumber of cameras 116, 118 coupled to a sensory processing system 120.Cameras 116, 118 can be any type of camera, including cameras sensitiveacross the visible spectrum or with enhanced sensitivity to a confinedwavelength band (e.g., the infrared (IR) or ultraviolet bands); moregenerally, the term “camera” herein refers to any device (or combinationof devices) capable of capturing an image of an object and representingthat image in the form of digital data. For example, line sensors orline cameras rather than conventional devices that capture atwo-dimensional (2D) image can be employed. The term “light” is usedgenerally to connote any electromagnetic radiation, which may or may notbe within the visible spectrum, and may be broadband (e.g., white light)or narrowband (e.g., a single wavelength or narrow band of wavelengths).Cameras 116, 118 are preferably capable of capturing video images (i.e.,successive image frames at a constant rate of at least 15 frames persecond); although no particular frame rate is required. The capabilitiesof cameras 116, 118 are not critical to the technology disclosed, andthe cameras 116, 118 can vary as to frame rate, image resolution (e.g.,pixels per image), color or intensity resolution (e.g., number of bitsof intensity data per pixel), focal length of lenses, depth of field,etc. The cameras 116, 118 need to have a higher resolution than thefeatures of the calibration image. In one implementation, the resolutionof the cameras 116, 118 is similar to the resolution of the AR headsetdisplay resolution. Additionally, the cameras 116, 118, in animplementation, have a field of view that is high (wide) enough tocapture, alone or in combination, the entirety of the images reflectedoff of the reflectors 106, 108. In general, for a particularapplication, any cameras capable of focusing on objects within a spatialvolume of interest can be used.

Sensory processing system 120, which can be, e.g., a computer system,can control the operation of cameras 116, 118 to capture images of theregion of interest.

The system 114 can also include sensors 122, 124 to capture motions ofthe AR headset 100 or other components and/or individual relatedthereto. Information from sensors 122, 124 can be applied to models ofimages taken by cameras 116, 118 to cancel out the effects of motions ofthe AR headset 100, providing greater accuracy to the augmented realityexperience rendered by AR headset 100. For some implementations, anothervideo input device can be utilized to generate a digitized video imageof the images. The cameras 116, 118 can be any digital device that isdimensioned and configured to capture still or motion pictures and toconvert those images to a digital stream of information that can bemanipulated by a computer. For example, cameras 116, 118 can be digitalstill cameras or digital video cameras, and the like.

Continuing with the description of system 1000, presentation interface126 employs presentation techniques in conjunction with the sensorybased tracking in order to present virtual (or virtualized real) objects(visual, audio, haptic, and so forth) created by applications loadableto, or in cooperative implementation with, the AR headset 100 to providea user of the device with a personal augmented reality experience.Presentation can include an image or other visual representation of anobject. The presentation interface 126 is what drives the displays 102,104 to display images and can also use the distortion mapping transforms(as obtained from the calibration performed after the manufacturing tocompensate for manufacturing deviations and as obtained fromcalibrations performed while the user is interacting with the ARenvironment) to adjust images displayed by the displays 102, 104.

As briefly discussed above, calibration and alignment are morechallenging for AR headsets with a wide FoV, because they have opticsthat create a stereo-divergent off-axis distortion that cannot beaccurately modelled with conventional radial polynomials. This highstandard of accuracy requires a distortion model that represents thephysical geometry of the optical system.

While modern small-scale prototyping techniques for AR headsets provideexcellent price efficiency, this comes at the cost of lower mechanicaltolerancing, relative to the requirements of near-eye optical systems.Distortions are locally linear, but as the FoV increases, the amount ofnon-linearity a single continuous distortion function needs to coverincreases substantially. It is necessary to measure mechanicaldeviations of the optical elements to create a valid distortion mappingtransform.

In optics, modeling an optical system by raytracing is very effective.Raytracing makes it possible to find the mapping between eye space andscreen space. Raytracing refers to the process of tracing the path raysof light travel from the light source, through the optical system, tothe eye. That is, raytracing is a rendering technique for generating animage by tracing the path of light as pixels in an image plane andsimulating the effects of its encounters with virtual objects.Raytracing makes it possible to simulate where a given ray of lightentering the eye came from on the display, so one can precisely map thedistortion between the eye and the screen. By measuring where an eye raylands on the screen, one can construct a distortion mesh whichrepresents how the image needs to be distorted on the screen tocompensate for the distortion of the optical system and achieve arectilinear image again. This distortion is the inverse of thedistortion of the optical system. For computational convenience, theserays are typically cast out backwards—from the eye towards the screen,since light travels along the same path regardless of its direction.This allows the system to only consider the rays of light that interactwith the eye. Raytracing makes it possible to parameterize distortion interms of physical elements.

Raytracing is inexpensive and effective when optics are known. However,3D printing of AR headsets and ergonomics make optics unknown. Bycapturing images on a monitor through the AR headset 100, the optics canbe defined, which makes possible real time IPD adjustment and eyetracking, in addition to the disclosed methods for compensating fordistortions caused by manufacturing deviations in the AR headset 100. Bycombining the raytracing model with eye tracking, one can compensate forthese inconsistencies in real time.

Additionally, reference element 128 illustrates the AR headset 100placed on the head of a user.

FIG. 2 illustrates a block diagram of an AR calibration rig 200 thatincludes a calibration and alignment system 201, and an external displayscreen 202. FIG. 2 also illustrates a top view of the AR headset 100, asillustrated in FIG. 1 that is to be calibrated using the AR calibrationrig 200. The AR headset 100 includes displays 102, 104 and reflectors106, 108, as previously discussed with reference to FIG. 1. The ARcalibration rig 200 also includes cameras 204, 206, which act as theuser's eyes for purposes of calibrating the AR headset 100. The ARcalibration rig 200 may include more cameras or may use a single camera.Further, the AR calibration rig 200 also includes a stationary stand(not illustrated) for holding the AR headset 100 at a specific heightand specific distance from the cameras 204, 206, so that calculationsperformed by the calibration and alignment system 201 can be based onknown distances and heights. Additionally, the cameras 204, 206 areplaced at a distance from the reflectors 106, 108 so as to replicate adistance of the user's eyes. This replicated distance can be determinedbased on an optimum known distance from the reflectors 106, 108 or adistance that is suitable for most users of the AR headset 100. Thecameras 204, 206 can also be adjusted to replicate a desired IPD 210.The desired IPD 210 can be based on an optimum known IPD or an IPD thatis within a generically known average of IPDs.

The calibration and alignment system 200 includes a calibration imagegenerator 212, an optimization engine 214 and a database that storesdistortion mapping transforms 216. The calibration image generator 212provides a calibration image to the external display screen 202 that isexternal to the AR headset 100 and that is viewable through thereflectors 106, 108 of the AR headset 100. The calibration imagegenerator 212 also provides an inverse of the calibration image (i.e.,an inverse calibration image) to the displays 102, 104 of the AR headset100, such that the inverse calibration image is reflected off of therespective reflectors 106, 108 and observed by cameras 204, 206 of theAR calibration rig 200 while the cameras 204, 206 are simultaneouslyobserving the calibration image on the external display screen 202. Asmentioned above, cameras 204, 206 of the AR calibration rig 200 arelocated inside the AR headset 100 at a location representing a user'spoint of view.

Continuing the description of FIG. 2, cameras 204, 206 of the ARcalibration rig 200 are aligned to a position within AR headset 100 thatallows the cameras 204, 206 to observe, from an optimal viewinglocation, the calibration image projected onto reflectors 106, 108 ofthe AR headset 100. The optimal view location can be determined based ona location of a foci, such as Foci 1 258 illustrated in FIG. 2.Ellipsoids are defined by a pair of Foci and the length of their MinorAxis. Reflector ellipsoid 228 shows the optimal view location, Foci 1258. The light that comes from Foci 2 238, which one of the displays102, 104 is in line with, will converge on Foci 1 258. In anotherimplementation, a different reflector design with a different reflectorshape, even aspheric reflectors, can be utilized with the disclosedcalibration system. The optimal view location can be determined in otherways that will be apparent to a person of ordinary skill in the art andcan be based on, for example, observed or obtained user information,average IPD, etc.

The optimization engine 214 receives images captured by the cameras 204,206 and creates a distortion mapping transform by implementing analgorithm to search through various projection positions of the inversecalibration image provided to the displays 102, 104 of the AR headset100 until the inverse calibration image observed by each camera 204, 206cancels out an acceptable portion of the calibration image provided toexternal display screen 202 as observed, by the cameras 204, 206 throughthe reflectors 106, 108. The optimization engine 214 is able to createthe distortion mapping transform to be individualized for each of theuser's eyes (i.e., individualized for each of the cameras 204, 206) byindependently searching through the various projection positions of theinverse calibration image as provided to and projected by the displays102, 104. The distortion mapping transforms 218 store the transformscreated by the optimization engine 214. In a different implementationcalibration and alignment system 201 can utilize a differentorganization of functional blocks that result in storing distortionmapping transforms 218 for each distinct AR headset 100 that iscalibrated by the AR calibration rig 200.

The AR calibration rig 200 is configured to essentially peer through anoptical system (e.g., the AR headset 100) to gain insight into thealignment of the optical system. By comparing the virtual image (e.g.,inverse calibration image) against a real-world reference (e.g., thecalibration image) it is possible to measure an aggregate deviation ofthe components in the optical system (e.g., the AR headset 100). Byusing a class of algorithms called numerical optimizers, it is able tosolve for the configuration of optical components that minimizes thedistortion mismatch between the real-world reference (e.g., thecalibration image displayed by the external display screen 202) and thevirtual image (e.g., the inverse calibration image projected by thedisplays 102, 104). These algorithms are implemented by the optimizationengine 214. As the optimization engine 214 implements the algorithms itthen provides transform information to apply to the inverse calibrationimage and/or provides an updated inverse calibration image to thecalibration image generator 212 to be provided to the displays 102, 104.These algorithms essentially “jiggle” the inverse calibration imagesprovided to the displays 102, 104 until the reflected calibration imagescancel out a sufficient amount of the calibration image displayed by theexternal display screen 202. This “jiggling” of the inverse calibrationimages is a simplified explanation of the algorithms. More complexalgorithms can be implemented, such as gradientless optimizationalgorithms (e.g., a Nelder-Mead algorithm). Further, a neural networkcan be trained and implemented to perform the optimization.

Moreover, the calibration and alignment system 201 handles opticalraytracing and 3D rendering. Calibration and alignment system 200triangulates the position and orientation of external display screen 202relative to the stationary testing base of the AR headset 100,displaying a pattern with a sequence of dots on the external displayscreen 202. The dots are detected and triangulated individually, whichallows for the six degrees of freedom positioning of the externaldisplay screen 202 with respect to the AR headset 100 and/or the cameras204, 206.

Calibration image generator 212 renders the inverse of the calibrationimage displayed on external display screen 202 so as to be displayed bythe displays 102, 104 and projected to the reflectors 106, 108 in thesame position as the calibration image is perceived by the cameras 204,206 through the reflectors 106, 108. If the inverse calibration imagesreflected by the reflectors 106, 108 match up perfectly with thecalibration image displayed by the external display screen 202, thecalibration image displayed by the external display screen 202 will beadditively canceled out to uniform white as perceived through thecameras 204, 206. The optimization engine 214 can measure the deviationfrom perfect white as the distortion error caused by the manufacturingdeviations of the physical optical system of the AR headset 100. Notethat the external display screen 202 and the AR headset 100 each havetheir own non-linear gamma curves which negate the additive whitephenomenon in theory but have a negligible effect in practice.

Turning back to the optimization algorithms implemented by theoptimization engine 214, optimization engine 214 runs a gradientlesssimplex optimizer system-in-the-loop to independently search throughvarious projection positions of the inverse calibration image providedto the displays 102, 104 until the inverse calibration image(s) observedby the cameras 204, 206 cancel out an acceptable portion of thecalibration image provided to external display screen 202 as observedthrough the reflectors 106, 108 by the cameras 204, 206. In oneimplementation, the system-in-the-loop gradient-less optimizer is aNelder-Mead optimizer to converge on an acceptable optical configurationwith a very high degree of precision.

As illustrated in FIG. 2, the AR calibration rig 200 includes twocameras 204, 205 (a right-eye camera and a left-eye camera). In anotherimplementation, there can be only a single camera. The single camera mayneed to be place further away such that it can observe reflections fromboth reflectors 106, 108. In another implementation, there can be morethan two cameras. For example, there can be two cameras for eachreflector (e.g., first and second right-eye cameras and first and secondright-eye cameras). The four cameras could be arranged, so that there isan upper and lower right-eye camera and an upper and lower left-eyecamera. Other camera configurations will be apparent to a person ofordinary skill in the art. Regardless of whether one, two or fourcameras are used, or some other configuration of cameras, theoptimization engine 214 implements the optimization algorithms based onthe inverse calibration image(s) and the calibration image observed bythe cameras to create the distortion mapping transforms 218. Once adistortion mapping transform is created for a particular AR headset 100,the distortion mapping transform can be loaded onto the AR headset 100prior to delivery to a retailer or an end user (e.g., the distortionmapping transform can be loaded onto the firmware of the AR headset100). Alternatively, the end user of the AR headset 100 can obtain thespecific distortion mapping transform by entering in a serial numberduring setup of the AR headset 100 and obtaining a download of thespecific distortion mapping transform based on the serial number. Otherways of obtaining the specific distortion mapping transform thatcorresponds to the specific AR headset will be apparent to a person ofordinary skill in the art.

When the calibration is implemented using two cameras allows thecalibration to observe the optics from each camera (or eye)simultaneously, allowing the technology disclosed to provide distortionmapping transforms resulting in an accurate optical system for each ARheadset 100 that can be raytraced from any perspective.

Additional implementations of the technology disclosed include (i) eachvertex of a distortion mesh observed by the cameras being independent,but traveling along a fixed set of optical elements, wherein thedistortion mesh can be recomputed in a vertex shader of a graphicalprocessing unit of the AR headset 100 and (ii) the distortion mesh beingmultithreaded and vectorized on a central processing unit of the ARheadset 100, resulting in optimizations coming in under a fifth of amillisecond per eye. Accordingly, it is possible to perform numericaloptimization without a gradient when the cost function is quick toevaluate (i.e., a “one-shot” photometric quality metric that is able tocalibrate using a single calibration image as opposed to structure lightor gray code mapping that would require multiple calibration images toget a similar fit quality estimate) and there are few free-parameters.Nelder-Mead is the most popular of these gradientless optimizers due toits performance and simplicity of implementation. However, a person ofordinary skill in the art would understand that other types ofgradientless optimizers can be implemented. Furthermore, a benefit ofusing the inverted calibration image that is essentially overlaid ontothe calibration image, is that it is possible to use additive colorsynthesis to create a dense photometric cost function that allows forthis ‘one-shot’ (i.e., one image) estimation of the quality of thedistortion correction.

These same calibration techniques can be used to update the distortionmapping transform generated using the AR calibration rig 200 duringactual use of the AR headset 100 by an end user. As discussed earlier,the AR headset 100 can move and shift during use, necessitating someintelligent distortion correction that can compensate for the moving andshifting, and that can take into account the user's head motion as wellas differences in head shape, head size and IPDs of various users. Inother words, the calibration performed by the calibration rig 200 cancompensate for manufacturing deviations by calibrating the AR headset100 based on the view point of a “generic” user or an “average” user,but it cannot compensate for different users having different physicalfeatures and for the moving and the shifting of the AR headset 100during use.

This “during use” calibration can be performed by contemporaneouslyupdating the distortion mapping transform implemented by the AR headset100 while the user is in an AR experience. Specifically, the “duringuse” calibration can be performed for each display frame of the ARexperience by implementing the cameras 116, 118 (see FIG. 1) and/orsensors 122, 124 (see FIG. 1) to perform eye tracking to identify (i) alocation of one or more of the user's eyes with respect to thereflectors 106, 108 and or display 102, 104 of the AR headset 100 and/or(ii) a viewing direction of one or more of the user's eyes with respectto the reflectors 106, 108 and or display 102, 104 of the AR headset100. After the performing of the eye tracking, the technology disclosedcan, for each display frame, update the distortion mapping transform tocompensate for (i) a change in the identified location of either of theuser's eyes with respect to the reflectors 106, 108 and or display 102,104 of the AR headset 100 and/or (ii) a change in the identified viewingdirection of either of the user's eyes with respect to the reflectors106, 108 and or display 102, 104 of the AR headset 100. Once thedistortion mapping transform is updated, the updated distortion mappingtransform can be implemented in a next displayed frame of the ARexperience. Accordingly, the AR headset 100 can be continuouslyself-calibrating during the user's AR experience. This “during use”calibration essentially performs physical modelling by tracking thelocation and/or viewing direction of one or more of the user's eyes.This physical modelling allows for tracking 3D pupil position to adjustthe distortion in real-time to compensate for eye movements and movementand shifting of the AR headset 100.

When using two cameras to perform eye tracking it is possible tooptimize/calibrate for an optical configuration that appears correctwhen raytraced from two perspectives for each eye. Physical modelling isbetter at the “tails” and fitting a physical model with a properfunction allows for extrapolation and better behavior at extreme ends ofthe “fit.” In contrast, polynomial “fits” often quickly diverge andbecome unreasonable outside of measured data.

As an alternative to the black and white calibration image illustratedin the various figures of this application, gray-code patterns can beused on both the external display screen 202 and the AR headset 100.This can allow for saving a set of ray-direction-mappings to optimizeagainst with a gradient-descent optimizer, using the gradients from theray tracer. This alternative approach may improve both the speed andprecision of the calibration.

FIG. 3A illustrates an example calibration image as displayed by anexternal display screen and FIG. 3B illustrates an example invertedcalibration image as projected onto left and right reflectors of an ARheadset.

Specifically, FIG. 3A illustrates the calibration image 300 as displayedby, for example, the external display screen 206 of FIG. 2. Variousgraphics overly the calibration image 300 of FIG. 3A. These variousgraphics are not part of the calibration image 300, but ratherillustrate the field of view and perspective of the AR headset while thecalibration image 300 is being displayed by the external display screen206. For example, the red dots illustrated in FIG. 3A that areoverlaying the calibration image 300 represent the triangulated points,from the above-described triangulation, which constrain the virtualmonitor (e.g., constrain the cameras 204, 206 with respect to theexternal display screen 202) in space. The green lines represent theedges of the ellipsoids, which the (ellipsoidal) reflectors 106, 108 area cross section of. The blue dots represent the principal points of thecameras 204, 206. The white dots represent the current perspective fromwhich the ray tracer is casting its rays, to create the distortioncompensation. The two red dots near the bottom of FIG. 3A represent theInner Foci of each of the ellipsoids.

FIG. 3B illustrates an inverse of calibration image 300 as reflected byreflectors of the AR headset. Specifically, FIG. 3B illustrates inversecalibration image 302, as projected onto a left-eye reflector of an ARheadset and illustrates an inverse calibration image 304, as projectedonto a right-eye reflector of the AR headset. The inverse calibrationimages 302, 304 are both the inverse of calibration image 300, eventhough they appear as two different images. As can be seen from FIGS. 3Aand 3B, only portions of the inverse calibration images 302, 304 areactually reflected by the reflectors of the AR headset. In other words,inverse calibration image 302 that is presently reflected is only aportion of the inversion of the entire calibration image 300 and inversecalibration image 304 that is presently reflected in only a portion ofthe inversion of the entire calibration image 300. The inversecalibration images 302, 304 may be reflected such that overlappingportions of the inversion of the calibration image 300 are reflected onboth the right-eye reflector of the AR headset and the left-eyereflector of the AR headset.

Additionally, as can be seen by comparing the calibration image of FIG.3A and the inverse calibration images 302, 304 of FIG. 3B, the inversecalibration images 302, 304 do not match up with the orientation andperspective of the calibration image 300. As such, when the inversecalibration images 302, 304 are viewed by cameras (or by the user) theinverse calibration images 302, 304 will not cancel out the calibrationimage 300 as also viewed by the cameras (or by the user). This meansthat the AR headset is not calibrated very well.

FIG. 4A illustrates an example inverse calibration image as reflected byleft-eye and right-eye reflectors of an AR headset before calibrationand FIG. 4B illustrates an example inverted calibration image asreflected by left-eye and right-eye reflectors of the AR headset aftercalibration.

Specifically, FIG. 4A illustrates inverse calibration image 400, asreflected by the left-eye reflector of the AR headset and alsoillustrates inverse calibration image 402, as reflected by the right-eyereflector of the AR headset. Inverse calibration images 400, 402 are thesame as inverse calibration images 302, 304 of FIG. 3B. FIG. 4Billustrates, after a calibration is performed, inverse calibration image404, as reflected by the left-eye reflector of the AR headset andillustrates, after calibration is performed, inverse calibration image406, as reflected by the right-eye reflector of the AR headset. As canbe seen by respectively comparing inverse calibration images 400, 402 ofFIG. 4A and inverse calibration images 404, 406 of FIG. 4B, the inversecalibration images 404, 406 have been shifted so that they more closelyalign with the calibration image 300 displayed by the external displayscreen 206.

FIG. 5A illustrates an example calibration image as displayed by anexternal display screen and FIG. 5B illustrates an example invertedcalibration image as projected onto left and right reflectors of an ARheadset after a calibration has been performed.

Similar to FIG. 3A, FIG. 5A illustrates the calibration image 500 asdisplayed by, for example, the external display screen 206 of FIG. 2.Various graphics overly the calibration image 500 of FIG. 5A. Thesevarious graphics are not part of the calibration image 500, but ratherillustrate the field of view and perspective of the AR headset while thecalibration image 500 is being displayed by the external display screen206. For example, the red dots illustrated in FIG. 5A that areoverlaying the calibration image 500 represent the triangulated points,from the above-described triangulation, which constrain the virtualmonitor (e.g., constrain the cameras 204, 206 with respect to theexternal display screen 202) in space. The green lines represent theedges of the ellipsoids, which the (ellipsoidal) reflectors 106, 108 area cross section of. The blue dots represent the principal points of thecameras 204, 206. The white dots represent the current perspective fromwhich the ray tracer is casting its rays, to create the distortioncompensation. The two red dots near the bottom of FIG. 5A represent theInner Foci of each of the ellipsoids.

FIG. 5B illustrates an inverse of calibration image 500 as reflected byreflectors of the AR headset. Specifically, FIG. 5B illustrates inversecalibration image 502, as projected onto a left-eye reflector of an ARheadset after calibration and illustrates an inverse calibration image504, as projected onto a right-eye reflector of the AR headset aftercalibration. Further, as illustrated, the inverse calibration images502, 504 are closer to being aligned with (e.g., matching) correspondingportions of the calibration image 500, as opposed to the inversecalibration images 302, 304 being aligned with (e.g., matching)corresponding portion of the calibration image 500. This calibration, asperformed, compensates for any manufacturing deviations in the ARheadset than can cause distortions, as perceived by the user.

FIG. 6 illustrates actual before and after calibration images as viewedby a camera through a reflector of an AR headset. Specifically, FIG. 6illustrates an actual screen capture by, for example, the camera 122 ofFIGS. 1 and 2, taken while viewing, for example, both (i) thecalibration image 500 as viewed through the left-eye reflector of the ARheadset and (ii) the inverted calibration image 502 as reflected by theleft-eye reflector of the AR headset. Screen capture 600 is the view ofthe camera 122 before calibration and screen capture 602 is the view ofthe camera 122 after calibration. As clearly illustrated, screen capture600 includes a significant portion of the calibration image 500 overlaidby the inverted calibration image 502. In other words, the calibrationimage 500 and the inverted calibration image 502 do not cancel eachother out, which would result in the camera 122 observing “white space.”

In contrast, screen capture 602 has more “white space” as a result ofthe calibration, such that the calibrated inverted calibration image 502cancels out more of the calibration image 500. The calibrator (i.e., theperson and/or machine performing the calibration) can determine and seta threshold of how much of the calibration image 500 must be cancelledout by the inverted calibration image 502. This can be done by, forexample, measuring the amount of “white space” in the screen capture602.

FIGS. 7 and 8 list an example distortion mapping transform, expressed inJSON, that gets loaded onto the AR headset 100 or that is continuouslyupdated during an AR experience, as described above. This distortionmapping transform can be utilized by, for example, an AR headset engine(not illustrated) to compensate for distortions caused by manufacturingdeviations or by moving or shifting of the AR headset 100 during use orby differences in head size, head shape and/or IPD. In anotherimplementation the distinct distortion mapping transform gets stored inthe headset hardware. In yet another case, distortion mapping transformscan be stored in the cloud and delivered, based on the headset's serialnumber.

The distinct distortion mapping transform illustrated in FIGS. 7 and 8lists designations for the left-eye optics 700 and right-eye optics 702.With respect to the left-eye optics 700, a description of an ellipsoidalreflector's sizing, position and rotation are defined in‘ellipseMinorAxis’, ‘ellipseMajorAxis’, and ‘sphereToWorldSpace’variables (see lines 3, 4 and 32-49, respectively, of FIG. 7, forexample). Further, with respect to the left-eye optics 700, the distinctdistortion mapping transform also includes a description of the screen'stransform and sizing, in the ‘screenForward’, ‘screenPosition’, and‘worldToScreenSpace’ variables (see lines 5-9, 10-14 and 50-60,respectively, of FIG. 7, for example). Additionally, with respect to theleft-eye optics 700, the distinct distortion mapping transform furtherincludes a description of where the eye's assumed/default position andfield of view is, in the ‘eyePosition’, ‘eyeRotation’, and‘cameraProjection’ variables (see lines 15-19, 20-25 and 26-31,respectively, of FIG. 7, for example). Similar variables are listed forthe right-eye optics 702 in FIGS. 7 and 8.

FIG. 9 illustrates an example distortion mapping transform for aspecific headset, such as a SteamVR® headset that uses specificmetadata. For example, different headsets can utilize differentinformation (e.g., metadata) for calibration purposes and the distortionmapping transform can include these different sets of metadata. FIG. 9lists the example distortion mapping for the SteamVR® headset as aresult of performing a calibration.

FIG. 10 illustrates an example computing system that can comprise one ormore of the elements shown in FIGS. 1 and 2. In particular, FIG. 10illustrates an exemplary computing system 1000, such as a PC (or othersuitable “processing” system), that can comprise one or more of theelements shown in FIGS. 1-2 according to an implementation.

As shown, computer system 1000 comprises elements coupled viacommunication channels (e.g. bus 1001) including one or more general orspecial purpose processors 1002, such as a Pentium® or Power PC®,digital signal processor (“DSP”), or other processing. System 1000elements also include one or more input devices 1003 (such as a mouse,keyboard, joystick, microphone, remote control unit, tactile, biometricor other sensors, and so on), and one or more output devices 1004, suchas a suitable display, joystick feedback components, speakers, biometricor other actuators, and so on, in accordance with a particularapplication.

System 1000 elements also include a computer readable storage mediareader 1005 coupled to a computer readable storage medium 1006, such asa storage/memory device or hard or removable storage/memory media;examples are further indicated separately as storage device 1008 andnon-transitory memory 1009, which can include hard disk variants,floppy/compact disk variants, digital versatile disk (“DVD”) variants,smart cards, read only memory, random access memory, cache memory orothers, in accordance with a particular application. One or moresuitable communication devices 1007 can also be included, such as amodem, DSL, infrared, etc. for providing inter-device communicationdirectly or via suitable private or public networks, such as theInternet. Non-transitory (working) memory 1009 is further indicated asincluding an operating system (“OS”) 1091, predictive discrepancydeterminer 1013 and other programs 1092, such as application programs,mobile code, data, or other information for implementing variouselements of FIGS. 1 and 2, which might be stored or loaded thereinduring use.

System 1000 element implementations can include hardware, software,firmware or a suitable combination. When implemented in software (e.g.as an application program, object, downloadable, servlet, and so on, inwhole or part), a system 1000 element can be communicated transitionallyor more persistently from local or remote storage to memory forexecution, or another suitable mechanism can be utilized, and elementscan be implemented in compiled, simulated, interpretive or othersuitable forms. Input, intermediate or resulting data or functionalelements can further reside more transitionally or more persistently ina storage media or memory, (e.g. storage device 1008 or memory 1009) inaccordance with a particular application.

Certain potential interaction determination, virtual object selection,authorization issuances and other aspects enabled by input/outputprocessors and other element implementations disclosed herein can alsobe provided in a manner that enables a high degree of broad or evenglobal applicability; these can also be suitably implemented at a lowerhardware/software layer. Note, however, that aspects of such elementscan also be more closely linked to a particular application type ormachine, or might benefit from the use of mobile code, among otherconsiderations; a more distributed or loosely coupled correspondence ofsuch elements with OS processes might thus be more desirable in suchcases.

Particular Implementations

One implementation discloses a computer-implemented method of using anAR calibration system to compensate for distortions caused bymanufacturing deviations in an AR headset. The method comprisesproviding a calibration image to an external display screen that isexternal to the AR headset and that is viewable through a reflector ofthe AR headset and providing an inverse of the calibration image to adisplay of the AR headset, such that the inverse calibration image isreflected off of the reflector and observed by a camera of the ARcalibration system while the camera is simultaneously observing thecalibration image on the external display screen. The camera of the ARcalibration system is located inside or within viewing distance of theAR headset at a location representing a user's point of view, and thecamera of the AR calibration system is aligned to a position that allowsthe camera to observe, from an optimal viewing location, the inversecalibration image projected onto the reflector of the AR headset. Themethod also includes creating a distortion mapping transform byimplementing an algorithm to search through various projection positionsof the inverse calibration image provided to the display of the ARheadset until the inverse calibration image observed by the cameracancels out an acceptable portion of the calibration image provided tothe external display screen as observed through the reflector by thecamera. The method further includes providing the distortion mappingtransform for implementation by the AR headset when an end user beginsan AR experience in order to compensate for the distortions caused bymanufacturing tolerances.

The method described in this section and other sections of thetechnology disclosed can include one or more of the following featuresand/or features described in connection with additional methodsdisclosed. In the interest of conciseness, the combinations of featuresdisclosed in this application are not individually enumerated and arenot repeated with each base set of features. The reader will understandhow features identified in this method can readily be combined with setsof base features identified as implementations.

For some implementations of the disclosed method, the AR headset islocated on a stationary testing base during the providing of the inversecalibration image.

For one implementation of the disclosed method, the AR headset includestwo reflectors, a right-eye reflector for a right-eye and a left-eyereflector for a left-eye, and the AR headset also includes two displays,a right-eye display and a left-eye display. The disclosed method furthercomprises providing the inverse calibration image to both the right-eyedisplay and the left-eye display, such that (i) the inverse calibrationimage is displayed by the right-eye display, reflected off of theright-eye reflector and observed by the camera while the camera issimultaneously observing the calibration image on the external displayscreen and (ii) the inverse calibration image is displayed by theleft-eye display, reflected off of the left-eye reflector and observedby the camera while the camera is simultaneously observing thecalibration image on the external display screen.

For one implementation of the disclosed method, the AR calibrationsystem includes two cameras, a right-eye camera and a left-eye camera,and the method further comprises aligning the right-eye camera to aposition within the AR headset that allows the right-eye camera toobserve, from an optimal right-eye viewing location, the inversecalibration image projected onto the right-eye reflector of the ARheadset, and aligning the left-eye camera to a position within the ARheadset that allows the left-eye camera to observe, from an optimalleft-eye viewing location, the inverse calibration image projected ontothe left-eye reflector of the AR headset. The inverse calibration imageis displayed by the right-eye display, reflected off of the right-eyereflector and observed by the right-eye camera while the right-eyecamera is simultaneously observing the calibration image on the externaldisplay screen. The inverse calibration image is displayed by theleft-eye display, reflected off of the left-eye reflector and observedby the left-eye camera while the left-eye camera is simultaneouslyobserving the calibration image on the external display screen. Themethod further comprises creating the distortion mapping transform byimplementing the algorithm to independently search through the variousprojection positions of the inverse calibration image provided to (i)the right-eye display until the inverse calibration image observed bythe right-eye camera cancels out an acceptable portion of thecalibration image provided to the external display screen as observedthrough the right-eye reflector by the right-eye camera and (ii) theleft-eye display until the inverse calibration image observed by theleft-eye camera cancels out an acceptable portion of the calibrationimage provided to the external display screen as observed through theleft-eye reflector by the left-eye camera.

For another implementation of the disclosed method, the AR calibrationsystem includes four cameras, a first right-eye camera, a secondright-eye camera, a first left-eye camera and a second left-eye camera.The method further comprises aligning the first right-eye camera to aposition within the AR headset that allows the first right-eye camera toobserve, from an optimal first right-eye viewing location, the inversecalibration image projected onto the right-eye reflector of the ARheadset. The method also includes aligning the second right-eye camerato a position within the AR headset that allows the second right-eyecamera to observe, from an optimal second right-eye viewing location,the inverse calibration image projected onto the right-eye reflector ofthe AR headset. The method further includes aligning the first left-eyecamera to a position within the AR headset that allows the firstleft-eye camera to observe, from an optimal first left-eye viewinglocation, the inverse calibration image projected onto the left-eyereflector of the AR headset, and aligning the second left-eye camera toa position within the AR headset that allows the second left-eye camerato observe, from an optimal second left-eye viewing location, theinverse calibration image projected onto the left-eye reflector of theAR headset. The inverse calibration image is displayed by the right-eyedisplay, reflected off of the right-eye reflector and observed by thefirst right-eye camera and the second right-eye camera while the firstright-eye camera and the second right-eye camera are simultaneouslyobserving the calibration image on the external display screen. Theinverse calibration image is displayed by the left-eye display,reflected off of the left-eye reflector and observed by the firstleft-eye camera and the second left-eye camera while the first left-eyecamera and the second left-eye camera are simultaneously observing thecalibration image on the external display screen. The disclosed methodfurther comprises creating the distortion mapping transform byimplementing the algorithm to independently search through variousprojection positions of the inverse calibration image provided to (i)the right-eye display until the inverse calibration image observed bythe first right-eye camera cancels out an acceptable portion of thecalibration image provided to the external display screen as observedthrough the right-eye reflector by the first right-eye camera, (ii) theright-eye display until the inverse calibration image observed by thesecond right-eye camera cancels out an acceptable portion of thecalibration image provided to the external display screen as observedthrough the right-eye reflector by the second right-eye camera, (iii)the left-eye display until the inverse calibration image observed by thefirst left-eye camera cancels out an acceptable portion of thecalibration image provided to the external display screen as observedthrough the left-eye reflector by the first left-eye camera and (iv) theleft-eye display until the inverse calibration image observed by thesecond left-eye camera cancels out an acceptable portion of thecalibration image provided to the external display screen as observedthrough the left-eye reflector by the second left-eye camera. For someimplementations, the first right-eye camera is an upper right-eyecamera, the second right-eye camera is a lower right-eye camera, thefirst left-eye camera is an upper left-eye camera and the secondleft-eye camera is a lower left-eye camera.

For some implementations of the disclosed method, the AR calibrationsystem includes the implemented algorithm triangulates a position of thecamera with respect to the reflector and the display, such that thetriangulated position of the camera is used to create the distortionmapping transform.

For some implementations of the disclosed method, the algorithm searchesthrough various projection positions of the inverse calibration imageusing a system-in-the-loop gradient-less optimizer. In oneimplementation of the method, the system-in-the-loop gradient-lessoptimizer is a Nelder-Mead optimizer.

For some implementations of the disclosed method, the distortion mappingtransform is stored in the AR headset for implementation during the ARexperience of the user. In other cases, the distortion mapping transformis stored in a database for later use and each distinct AR headsetaccesses a distinct distortion mapping transform stored relative to theserial number of the AR headset.

Another implementation includes the disclosed method ofcontemporaneously updating a distortion mapping transform implemented byan AR headset while a user is in an AR experience. The method comprisesfor each video frame of the AR experience: performing eye tracking toidentify (i) a location of both eyes of the user with respect to areflector and/or a display of the AR headset and (ii) a viewingdirection of both eyes of the user with respect to the reflector and/orthe display of the AR headset. The method also comprises updating thedistortion mapping transform to compensate for (i) a change in alocation of either of the user's eyes with respect to the reflectorand/or the display of the AR headset and (ii) a change in a viewingdirection of either of the user's eyes with respect to the reflectorand/or the display. The method further comprises implementing theupdated distortion mapping transform in a next display frame of the ARexperience. For some implementations of the disclosed method, apre-calibrated distortion mapping transform is implemented by the ARheadset prior to the AR experience and the updating of the distortionmapping transform initially updates the pre-calibrated distortionmapping transform.

Other implementations of the disclosed technology described in thissection can include a tangible non-transitory computer readable storagemedium having computer program instructions recorded thereon, thecomputer program instructions for using an augmented reality (AR)calibration system to compensate for distortions caused by manufacturingdeviations in an AR headset and the computer program instructions, whenexecuted on a processor, cause the processor to perform any of themethods described above. For one implementation of the non-transitorycomputer-readable recording medium, a pre-calibrated distortion mappingtransform is implemented by the AR headset prior to the AR experienceand the updating of the distortion mapping transform initially updatesthe pre-calibrated distortion mapping transform.

Another implementation of the disclosed technology described in thissection can include an AR calibration system including a camera, anexternal display screen, a stationary testing base, one or moreprocessors coupled to memory, the memory loaded with computer programinstructions to compensate for distortions caused by manufacturingdeviations in a AR headset, the computer program instructions, whenexecuted on the one or more processors, implement actions comprisingproviding a calibration image to the external display screen that isexternal to the AR headset and that is viewable through a reflector ofthe AR headset and providing an inverse of the calibration image to adisplay of the AR headset, such that the inverse calibration image isreflected off of the reflector and observed by the camera of the ARcalibration system while the camera is simultaneously observing thecalibration image on the external display screen. The camera of the ARcalibration system is located inside or within viewing distance of theAR headset at a location representing a user's point of view, and thecamera of the AR calibration system is aligned to a position that allowsthe camera to observe, from an optimal viewing location, the inversecalibration image projected onto the reflector of the AR headset. Theactions also comprise creating a distortion mapping transform byimplementing an algorithm to search through various projection positionsof the inverse calibration image provided to the display of the ARheadset until the inverse calibration image observed by the cameracancels out an acceptable portion of the calibration image provided tothe external display screen as observed through the reflector by thecamera, and providing the distortion mapping transform forimplementation by the AR headset when an end user begins an ARexperience in order to compensate for the distortions caused bymanufacturing tolerances.

Yet another implementation of the disclosed technology described in thissection can include an AR calibration system including a camera, anexternal display screen, a stationary testing base, one or moreprocessors coupled to memory, the memory loaded with computer programinstructions to update a distortion mapping transform implemented by anAR headset while a user is in an AR experience. The computer programinstructions, when executed on a processor, cause the processor toimplement actions comprising, for each video frame of the AR experience:performing eye tracking to identify (i) a location of both eyes of theuser with respect to a reflector and/or a display of the AR headset and(ii) a viewing direction of both eyes of the user with respect to thereflector and/or the display of the AR headset. Additional actionsinclude updating the distortion mapping transform to compensate for (i)a change in a location of either of the user's eyes with respect to thereflector and/or the display of the AR headset and (ii) a change in aviewing direction of either of the user's eyes with respect to thereflector and/or the display, and implementing the updated distortionmapping transform in a next display frame of the AR experience. Thedisclosed system is operable to perform any of the methods describedabove. In one implementation of the system, a pre-calibrated distortionmapping transform is implemented by the AR headset prior to the ARexperience and the updating of the distortion mapping transforminitially updates the pre-calibrated distortion mapping transform.

The preceding description is presented to enable the making and use ofthe technology disclosed. Various modifications to the disclosedimplementations will be apparent, and the general principles definedherein may be applied to other implementations and applications withoutdeparting from the spirit and scope of the technology disclosed. Thus,the technology disclosed is not intended to be limited to theimplementations shown but is to be accorded the widest scope consistentwith the principles and features disclosed herein. The scope of thetechnology disclosed is defined by the appended claims.

What is claimed is:
 1. A method of using an augmented reality (AR)calibration system to compensate for distortions caused by manufacturingdeviations in an AR headset, the method comprising: providing acalibration image to an external display screen that is external to theAR headset and that is viewable through a reflector of the AR headset,providing an inverse of the calibration image to a display of the ARheadset, such that the inverse calibration image is reflected off of thereflector and observed by a camera of the AR calibration system whilethe camera is simultaneously observing the calibration image on theexternal display screen, wherein the camera of the AR calibration systemis located inside or within viewing distance of the AR headset at alocation representing a users point of view, and wherein the camera ofthe AR calibration system is aligned to a position that allows thecamera to observe, from an optimal viewing location, the inversecalibration image projected onto the reflector of the AR headset;creating a distortion mapping transform by implementing an algorithm tosearch through various projection positions of the inverse calibrationimage provided to the display of the AR headset until the inversecalibration image observed by the camera cancels out an acceptableportion of the calibration image provided to the external display screenas observed through the reflector by the camera; and providing thedistortion mapping transform for implementation by the AR headset whenan end user begins an AR experience in order to compensate for thedistortions caused by manufacturing tolerances.
 2. The method of claim1, wherein the AR headset is located on a stationary testing base duringthe providing of the inverse calibration image.
 3. The method of claim1, wherein the AR headset includes two reflectors, a right-eye reflectorfor a right-eye and a left-eye reflector for a left eye, wherein the ARheadset includes two displays, a right-eye display and a left-eyedisplay, and wherein the method further comprises: providing the inversecalibration image to both the right-eye display and the left-eyedisplay, such that (i) the inverse calibration image is displayed by theright-eye display, reflected off of the right-eye reflector and observedby the camera while the camera is simultaneously observing thecalibration image on the external display screen and (ii) the inversecalibration image is displayed by the left-eye display, reflected off ofthe left-eye reflector and observed by the camera while the camera issimultaneously observing the calibration image on the external displayscreen.
 4. The method of claim 3, wherein the AR calibration systemincludes two cameras, a right-eye camera and a left-eye camera, whereinthe method further comprises: aligning the right-eye camera to aposition within the AR headset that allows the right-eye camera toobserve, from an optimal right-eye viewing location, the inversecalibration image projected onto the right-eye reflector of the ARheadset; and aligning the left-eye camera to a position within the ARheadset that allows the left-eye camera to observe, from an optimalleft-eye viewing location, the inverse calibration image projected ontothe left-eye reflector of the AR headset, wherein the inversecalibration image is displayed by the right-eye display, reflected offof the right-eye reflector and observed by the right-eye camera whilethe right-eye camera is simultaneously observing the calibration imageon the external display screen, wherein the inverse calibration image isdisplayed by the left-eye display, reflected off of the left-eyereflector and observed by the left-eye camera while the left-eye camerais simultaneously observing the calibration image on the externaldisplay screen, and wherein the method further comprises creating thedistortion mapping transform by implementing the algorithm toindependently search through the various projection positions of theinverse calibration image provided to (i) the right-eye display untilthe inverse calibration image observed by the right-eye camera cancelsout an acceptable portion of the calibration image provided to theexternal display screen as observed through the right-eye reflector bythe right-eye camera and (ii) the left-eye display until the inversecalibration image observed by the left-eye camera cancels out anacceptable portion of the calibration image provided to the externaldisplay screen as observed through the left-eye reflector by theleft-eye camera.
 5. The method of claim 3, wherein the AR calibrationsystem includes four cameras, a first right-eye camera, a secondright-eye camera, a first left-eye camera and a second left-eye camera,wherein the method further comprises: aligning the first right-eyecamera to a position within the AR headset that allows the firstright-eye camera to observe, from an optimal first right-eye viewinglocation, the inverse calibration image projected onto the right-eyereflector of the AR headset; aligning the second right-eye camera to aposition within the AR headset that allows the second right-eye camerato observe, from an optimal second righteye viewing location, theinverse calibration image projected onto the right-eye reflector of theAR headset; aligning the first left-eye camera to a position within theAR headset that allows the first left-eye camera to observe, from anoptimal first left-eye viewing location, the inverse calibration imageprojected onto the left-eye reflector of the AR headset; and aligningthe second left-eye camera to a position within the AR headset thatallows the second left-eye camera to observe, from an optimal secondleft-eye viewing location, the inverse calibration image projected ontothe left-eye reflector of the AR headset, wherein the inversecalibration image is displayed by the right-eye display, reflected offof the right-eye reflector and observed by the first right-eye cameraand the second right-eye camera while the first right-eye camera and thesecond right-eye camera are simultaneously observing the calibrationimage on the external display screen, wherein the inverse calibrationimage is displayed by the left-eye display, reflected off of theleft-eye reflector and observed by the first left-eye camera and thesecond left-eye camera while the first left-eye camera and the secondleft-eye camera are simultaneously observing the calibration image onthe external display screen, and wherein the method further comprisescreating the distortion mapping transform by implementing the algorithmto independently search through various projection positions of theinverse calibration image provided to (i) the right-eye display untilthe inverse calibration image observed by the first right-eye cameracancels out an acceptable portion of the calibration image provided tothe external display screen as observed through the right-eye reflectorby the first right-eye camera, (ii) the right-eye display until theinverse calibration image observed by the second right-eye cameracancels out an acceptable portion of the calibration image provided tothe external display screen as observed through the right-eye reflectorby the second right-eye camera, (iii) the left-eye display until theinverse calibration image observed by the first left-eye camera cancelsout an acceptable portion of the calibration image provided to theexternal display screen as observed through the left-eye reflector bythe first left-eye camera and (iv) the left-eye display until theinverse calibration image observed by the second left-eye camera cancelsout an acceptable portion of the calibration image provided to theexternal display screen as observed through the left-eye reflector bythe second left-eye camera.
 6. The method of claim 5, wherein the firstright-eye camera is an upper right-eye camera, the second right-eyecamera is a lower right-eye camera, the first left-eye camera is anupper left-eye camera and the second left-eye camera is a lower left-eyecamera.
 7. The method of claim 1, wherein the implemented algorithmtriangulates a position of the camera with respect to the reflector andthe display, such that the triangulated position of the camera is usedto create the distortion mapping transform.
 8. The method of claim 1,wherein the algorithm searches through various projection positions ofthe inverse calibration image using a system-in-the-loop gradient-lessoptimizer.
 9. The method of claim 8, wherein the system-in-the-loopgradient-less optimizer is a Nelder-Mead optimizer.
 10. The method ofclaim 1, wherein the distortion mapping transform is stored in the ARheadset for implementation during the AR experience of the user.
 11. Anon-transitory computer-readable recording medium having computerprogram instructions recorded thereon, the computer program instructionsfor using an augmented reality (AR) calibration system to compensate fordistortions caused by manufacturing deviations in an AR headset and thecomputer program instructions, when executed on a processor, cause theprocessor to perform a method comprising: providing a calibration imageto an external display screen that is external to the AR headset andthat is viewable through a reflector of the AR headset, providing aninverse of the calibration image to a display of the AR headset, suchthat the inverse calibration image is reflected off of the reflector andobserved by a camera of the AR calibration system while the camera issimultaneously observing the calibration image on the external displayscreen, wherein the camera of the AR calibration system is locatedinside or within viewing distance of the AR headset at a locationrepresenting a users point of view, and wherein the camera of the ARcalibration system is aligned to a position that allows the camera toobserve, from an optimal viewing location, the inverse calibration imageprojected onto the reflector of the AR headset; creating a distortionmapping transform by implementing an algorithm to search through variousprojection positions of the inverse calibration image provided to thedisplay of the AR headset until the inverse calibration image observedby the camera cancels out an acceptable portion of the calibration imageprovided to the external display screen as observed through thereflector by the camera; and providing the distortion mapping transformfor implementation by the AR headset when and end user begins an ARexperience in order to compensate for the distortions caused bymanufacturing tolerances.
 12. The non-transitory computer-readablerecording medium of claim 11, wherein the AR headset is located on astationary testing base during the providing of the inverse calibrationimage.
 13. An augmented reality (AR) calibration system including acamera, an external display screen, a stationary testing base, one ormore processors coupled to memory, the memory loaded with computerprogram instructions to compensate for distortions caused bymanufacturing deviations in an AR headset, the computer programinstructions, when executed on the one or more processors, implementactions comprising: providing a calibration image to the externaldisplay screen that is external to the AR headset and that is viewablethrough a reflector of the AR headset, providing an inverse of thecalibration image to a display of the AR headset, such that the inversecalibration image is reflected off of the reflector and observed by thecamera of the AR calibration system while the camera is simultaneouslyobserving the calibration image on the external display screen, whereinthe camera of the AR calibration system is located inside or withinviewing distance of the AR headset at a location representing a userspoint of view, and wherein the camera of the AR calibration system isaligned to a position that allows the camera to observe, from an optimalviewing location, the inverse calibration image projected onto thereflector of the AR headset; creating a distortion mapping transform byimplementing an algorithm to search through various projection positionsof the inverse calibration image provided to the display of the ARheadset until the inverse calibration image observed by the cameracancels out an acceptable portion of the calibration image provided tothe external display screen as observed through the reflector by thecamera; and providing the distortion mapping transform forimplementation by the AR headset when an end user begins an ARexperience in order to compensate for the distortions caused bymanufacturing tolerances.
 14. The system of claim 13, wherein the ARheadset is located on a stationary testing base during the providing ofthe inverse calibration image.
 15. A method of contemporaneouslyupdating a distortion mapping transform implemented by an augmentedreality (AR) headset while a user is in an AR experience, the methodcomprising: for each display frame of the AR experience: performing eyetracking to identify a location of both eyes of the user with respect toa reflector and/or a display of the AR headset; updating the distortionmapping transform to compensate for a change in the identified locationof either of the user's eyes with respect to the reflector and/or thedisplay of the AR headset; and implementing the updated distortionmapping transform in a next display frame of the AR experience.
 16. Themethod of claim 15, wherein the performing of the eye trackingidentifies a viewing direction of both eyes of the user with respect tothe reflector and/or the display of the AR headset, and wherein theupdating of the distortion mapping transform compensates for a change inthe identified viewing direction of either of the user's eyes withrespect to the reflector and/or the display.
 17. The method of claim 15,wherein a pre-calibrated distortion mapping transform is implemented bythe AR headset prior to the AR experience and the updating of thedistortion mapping transform initially updates the pre-calibrateddistortion mapping transform.
 18. A non-transitory computer-readablerecording medium having computer program instructions recorded thereon,the computer program instructions for updating a distortion mappingtransform implemented by an augmented reality (AR) headset while a useris in an AR experience and the computer program instructions, whenexecuted on a processor, cause the processor to perform a methodcomprising: for each display frame of the AR experience: performing eyetracking to identify a location of both of the user's eyes with respectto a reflector and/or a display of the AR headset; updating thedistortion mapping transform to compensate for a change in theidentified location of either of the user's eyes with respect to thereflector and/or the display of the AR headset; and implementing theupdated distortion mapping transform in a next display frame of the ARexperience.
 19. The non-transitory computer-readable recording medium ofclaim 18, wherein a pre-calibrated distortion mapping transform isimplemented by the AR headset prior to the AR experience and theupdating of the distortion mapping transform initially updates thepre-calibrated distortion mapping transform.
 20. An augmented reality(AR) calibration system including a camera, an external display screen,a stationary testing base, one or more processors coupled to memory, thememory loaded with computer program instructions to update a distortionmapping transform implemented by an augmented reality (AR) headset whilea user is in an AR experience, the computer program instructions, whenexecuted on a processor, cause the processor to implement actionscomprising: for each display frame of the AR experience: performing eyetracking to identify a location of both eyes of the user with respect toa reflector and/or a display of the AR headset; updating the distortionmapping transform to compensate for a change in the identified locationof either of the user's eyes with respect to the reflector and/or thedisplay of the AR headset; and implementing the updated distortionmapping transform in a next display frame of the AR experience.